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[57] ABSTRACT 

A system and a method are described for selecting and 
initiating security services for communication across com- 
puter networks. Negotiation is facilitated between nodes of 
a computer network to determine whether a security service 
is to be used, and if so, what particular method of providing 
the security service will be used. Communication with a 
wide range of different computer systems using different 
methods of providing various security services is thereby 
facilitated. 

28 Claims, 9 Drawing Sheets 
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SYSTEM AND METHOD FOR and hardware devices, they have no ability to tell which 

NEGOTIATING SECURITY SERVICES AND process is creating the data they receive when it arrives at the 

ALGORITHMS FOR COMMUNICATION hardware interface. Thus, hardware-based encryption meth- 

ACROSS A COMPUTER NETWORK ods are typically inflexible in that either all network traffic 

5 must be encrypted or none of it 

Ordinary encryption makes reading messages nearly f 
BACKGROUND OF THE INVENTION impossible, but does not protect against changes to the dataJ 

1 Technical Field For data integrity another type of security service, known 

- as tamper detection or crypto-checksumming 
The present invention re ates to security services for l0 ^^J^^ hcl ^ s to cnsure ^ ^ has not been 
communications in a networked computer environment. In ^ ^> mmiJIlication ^ a computer network In 

particular, the present invention relates to the negotiation of checksumraingf ^ xn6c[ periodically interjects a special 
security services such as authentication, encryption, and ^ a mto a ^ stream being sent to the 

checksumming as weU as the selection of algorithms for reccivcr B exar^^o the checksum, the receiver can 
each particular security service. The present invention also 1S 4 cthcr ^ data has been modified during trans- 

relates to providing a key resizing facility for use with j^g^ 

multiple security services. As ^ encryption ^ a gotithlDS may also be used for 

2. Description of the Related Art checksumming. For example, MD5 has become an induslry- 

In a networked computer environment, various security standard algorithm for cryptographic checksumming, 
services for communications between network nodes may be 20 According to the MD5 algorithm, a checksum or message 
implemented. For example, security services such as digest is added to each packet during transmission. When (he 
authentication, encryption, and checksiimming may be pro- packets are received at the destination, they are immediately 
vided for communications across a computer network checked for data modifications. If a modification is 

Authentication is a security service which verifies that a discovered, the violation may be recorded in the system log 
network user is who he claims to be. Methods of providing files, and any operation in progress when tampering took 
authentication include those in which the receiver may query place may be immediately terminated, 
the socket; those using some authentication service, such as Although many of the security services described above 
a Kerberos server, and those using a cryptographic token M t commonly used in communications across computer 
which is explicitly passed across the connection. Other ^ networks, and users may sometimes be able to specify a 
methods of providing authentication include physical- particular security service, most network systems do not 
device-assisted authentication and biometric authentication. provide a choice as to which particular algorithm to use for 
In physical-device-assisted authentication, devices such as me specified security service. Computer security services 
"smart cards" may be used to provide a randomly generated, m typically "h aid- wired" into a computer system so that 
unpredictable access code. Biometric authentication relies ^ on iy a single method or algorithm for providing a security 
on a unique physical characteristic of the user, such as a service is available. 

finger print Some related art systems allow users to specify a security 

I ' Encryption is a type of security service by which com- service and algorithm on a "per message* 1 basis. For 
1 municatioQS over a network are encoded to help ensure example, a system may allow the sender to specify in each 
(privacy of sensitive data. The desired result is to scramble^ message header an algorithm for a security service. The 
the information in such a way that a person cannot read the \ receiver examines each message header to determine which 
message without an "encryption key." Thus, the encryptioiH security service and algorithm to use. The specification from 
key should only be known to the sender and receiver. the message sender is treated as an absolute demand for the 

Various algorithms may be used for encryption. For particular service and algorithm. If, for some reason, the 
example, RC4, developed by RSA Data Security, Inc., has 45 receiver cannot use the specified service or algorithm, the 
become widely used as a method for high speed encryption. communication fails. Such failure may occur, for example. 

LRC4 has the advantage of providing relatively good security if the receiver lacks the facilities to provide the specified 
with relatively low total performance degradation. The Date algorithm. Failure may also occur if the receiver of the 
Encryption Standard ( M DES")algorithm has been the U.S. message is under instructions not to use a particular algo- 
govemment standard for many years, and is sometimes 50 rithm even though it is physically able to do so. For example, 
mandated in the financial services industry. In most special- various countries regulate which encryption algorithms, and 
ized banking systems in use today, the DES algorithm is what key sizes, may be used within their boundaries. Failure 
used to protect large international monetary transactions. might also occur if the receiver is unwilling to accept the 
DES is available in a domestic version, sometimes referred performance degradation of the particular algorithm speci- 
to as the DES 56 algorithm, which uses a 56-bit encryption 55 fied by the sender. 

key. DES Is also available in modified versions, such as DES Where one party simply demands a particular service. 
40, which uses a 40-bit encryption key. communication may fail even if it would have been mutually 

The traditional method of encrypting network data is to satisfactory not to use the particular service. For example, 
add hardware boxes or boards to each of the nodes in a suppose a first party prefers that checksumming be used, but 
network Alternatively, encryption boards or boxes can be 60 does not consider it necessary. In prior art systems, the 
added to all of the wide-area links within a company, party's only choices would be to specify checksumming or 
typically by adding proprietary boards to routers or bridges. not If the party specifies checksuxnming and the other party 
Hardware-based encryption has the disadvantages of rela- is unable to provide checksumming, the communication will 
tively high cost and limited compatibility with various fail even though it would have been satisfactory for both 
hardware platforms as well as with various types of network 65 parties not to have used checksuniming. By providing only 
protocol. Moreover, since hardware devices sit at a low level a binary "take it or leave if choice, the parties may end up 
of the Network "stack." which is made up of software layers in a situation which is less than optimal. For example, if the 
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first party chooses not to specify checksumming, and the 
other party could have provided it checksumming will not 
be used even though to use it would have been possible and 
even preferred by the first party. 

Furthermore, under a binary "take it or leave it" structure, 
the communication may proceed in a less than optimal 
fashion or even fail when a preferred algorithm is specified 
by one party. For example, if the first party specifies a 
version of DBS encryption which the receiver cannot 
provide, the communication will fail even though another 
algorithm such as RC4 may have been acceptable to both the 
sender and the receiver. 

As computer networks continue.. to grow, computer secu- 
rity will be increasingly require^Tto ensure that confidential 
communications are not intercepted by any of the large 
number of users who could potentially intercept the com- 
mumcation^Furmermore, larger networks will increasingly 
be connecting different types of computer systems which 
often have different preferences for security services and 
algorithms. 

SUMMARY OF THE INVENTION 

For the foregoing reasons, it would be desirable in a 
networked computer environment to facilitate negotiations 
between computers to determine a mutually satisfactory 
selection of security services and algorithms. In particular, it 
would be desirable to provide a negotiation process where at 
least one party has the option of indicating a preference for 
a service or algorithm, although the service or algorithm 
may not be required by that party. 

It would further be desirable to provide in a networked 
computer environment the ability to choose between several 
different methods or algorithms for providing security ser- 
vices. This would advantageously allow flexibility for 
expanding networks which may connect different types of 
computers using different types of algorithms. This would 
also advantageously allow users to add their own algorithms 
which may not be available from the software or hardware 
platform supplier. 

Thus, according to the present invention, a system and 
method for selecting security services for communication 
across a computer network is provided. The invention allows 
computer nodes to express preferences for security services 
and to come to agreement on mutually satisfactory methods 
and algorithms for providing security services. 

More specifically, the invention provides for determining 
whether a security service will be used for communication 
between two nodes in a networked computer environment 
by selecting preferences for using a security service at each 
of two nodes, communicating at least one of the preferences 
between the nodes, comparing the preferences of each node, 
and deterniining if the security service should be used. 

The invention further provides for selecting a method for 
the security service by creating at each node a list of 
acceptable methods for the security service, communicating 
at least one of the lists of methods to the other node, and 
selecting a mutually satisfactory security method, if one 
exists, by comparing the lists of acceptable methods. 

BRIEF DESCRIPTION OF THE DRAWINGS 

These and other methods, structures, features, aspects, 
and advantages of the present invention, wQl become more 
readily apparent from the following detailed description, 
which should be read in conjunction with the accompanying 
drawings, in which: 
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FIG. 1 illustrates connections between a computer net- 
work and individual computer nodes; 

FIG. 2 illustrates one possible internal structure for a 
computer node; 
3 FIG. 3 illustrates a preferred embodiment of the network 
services architecture which provides network security func- 
tions; 

FIG. 4 illustrates steps required to perform authentication; 
10 FIG. S illustrates how the connect protocol operates in the 
preferred embodiment; 

FIG. 6 illustrates a part of the negotiation process for a 
security service according to one embodiment of the inven- 
tion; 

15 FIG. 7 is a table of outcomes for combinations of client 
and server preferences relating to whether or not a security 
service should be used, according to one embodiment of the 
invention; 

FIG. 8 is a table of outcomes for combinations of client 
20 and server preferences relating to whether or not a security 
service should be used, according to a preferred embodiment 
of the invention; 
FIG. 9 illustrates the second phase of the negotiation 
^ process, which selects a particular method or algorithm of 
providing a security service; 

FIG. 10 illustrates how a preferred embodiment deter- 
mines if the communication will proceed, and if so whether 
a security service will be used; 
M FIG. 11 illustrates a packet format according to a pre- 
ferred embodiment of the invention; and 

FIG. 12 illustrates how a security service is restored and 
reset during breaks in communication. 

35 DETAILED DESCRIPTION OF THE 

PREFERRED EMBODIMENTS 

The following detailed description of the preferred 
embodiments of this invention is meant to be illustrative 
only of the principles of this invention and is not intended to 

40 limit the invention to the embodiments described herein. 
Other embodiments of this invention, having broader 
application, will be obvious to those skilled in the art in view 
of this disclosure. 

45 The following detailed description refers to a computer 
node at one end of the communication as a "client" and a 
computer node at the other end as a "server.*' These labels 
simply distinguish between two ends of a communication 
and are in no way meant to restrict the invention to client- 

5Q server architectures. Thus, the labels "client" and "server" 
may apply interchangeably to any node on a computer 
network. 

FIG. 1 illustrates an example of a networked computer 
environment 1*0 in which a network 102 is connected to, 

55 and facilitates communication between, computer nodes 
104, 106, and 108. Although network 102 is shown to 
connect three nodes, this invention is equally applicable to 
a network which connects as few as two nodes, or as many 
as millions of nodes. Network nodes may exist in the same 

50 building or they may be geographically distributed, even 
across different continents. 

FIG. 2 illustrates one possible internal structure for a 
computer node. As shown, computer node 104 ordinarily 
comprises a processor 110 connected to a memory 112. 

63 Node 104 may include security devices 114. which may be 
used to facilitate a particular security service. For example, 
security devices 114 may include a magnetic card reader or 
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fingerprint verification facility. Node 104 may also have security service 124, may have to do some additional 

security support hardware 116 provided to facilitate security cleanup activity. Accordingly, an additional "connection 

services such as encryption and checksumining. The pro- completion" function may be provided for that purpose, 

cess or communicates with network 102 either directly, or After the services are connected, the dispatch layer allows 

indirectly through security support hardware. Other inter- 5 the calling process to directly access the subordinate ser- 

connections between the elements in FIG. 2 are of course y* lccs 

possible, and nodes may contain additional components not During disconnect the dispatch layer calls a disconnect 

illustrated in FIG 2. » function for each service. The disconnect functions are 

^ n r St ^Zl7J^% F re-entrant and can be called repeatedly without ill effect 

security support hardware 116. in ^ , f\ _ , . . 

FIG. 3 is a diagram of a network services architecture 10 g Coromunicatons component 156 provides a number of 

capable of providing network security functions according a facilities: (1) it insulates the services from method of 

preferred embodiment of the invention. The network ser- transport which is useful for moving from one means of 

vices architecture comprise network services umbrella layer communication to another, (2) it provides a consistent way 

120 which ordinarily contains one or more network security , t0 transmit and receive data between machines that use 

services. For example. FIG. 3 shows an encryption security 13 different byte layouts; (3) during transmission, communica- 

service 122, a data integrity security service 124, and an component 156 bundles transmissions from different 

authentication service 126 residing under network services services together, and when each portion of the connection 

umbrella layer 120. Above the network services umbrella P* 1 *^ " finished, it transmits the data to the other processes; 

layer 120 ordinarily resides some transport layer. An nn ™ d W during reception, communication component 156 

example of one such transport layer is SQL*Net® devel- 20 segregates incoming data by service, organizes it into sepa- 

oped by Oracle Corporation. Network serviccs umbrella rate queues for each service, and verifies mat me type of data 

layer 120 preferably is able to operate with alternative received is the type of data requested, 

transport layers so that applications from alternative suppli- FIG. 4 illustrates how an authentication security service 

ers may easily be supported by the security services may be engaged. According to a preferred embodiment the 

described herein. Although FIG. 3 depicts an enibodiment authentication service is handled differently than encryption 

having three security services, namely encryption, or checksumming. This is because authentication is typically 

checksumming, and authentication, in general any number performed periodically, whereas encryption and checksum- 

of security services, or other services, may be provided. For ming are either enabled or disabled for all data transmissions 

example, a data compression service might be provided, M during a connection between two nodes, 

which requires negotiation of a common choice of data The server may handle authentication in one of two ways, 

compression algorithm. If the client passes authentication information, the server 

Each network security service ordinarily contains one or validates it by calling an authentication adapter. However, if 

more adapters which connect the service to a particular the client does not pass authentication information and the 

method or algorithm for performing the service. As shown 33 communications protocol supports it. the server attempts to 

in FIG. 3, Encryption security service 122 comprises RC4 perform authentication implicitly, by querying the network 

adapter 130, DES 56 adapter 134, and DES 40 adapter 138. socket to verify the identity of the client 

which use RC4 algorithm 132. DES 56 algorithm 136. and As FIG. 4 illustrates, in step 160 each process determines 

DES 40 algorithm 140 respectively. Although three adapters if authentication is required or requested. If authentication is 

are shown in FIG. 3, in general any number of adapters and ^ required or requested, in step 162, the process allocates and 

algorithms may be used for each security service type. initializes a context structure for authentication. In step 164. 

Similarly, data integrity security service 124 comprises the process then commences the connect protocol. 

MD5 adapter 142 which uses MD5 algorithm 144, and FIG. 5 illustrates in further detail the connect protocol of 

Authorization security service 126 comprises Kerberos ste p according to a preferred embodiment of the 

adapter 146, and Biometric adapter 150. which use the 43 invention. As shown m FIG. 5, at step 166 each process first 

Kerberos and Biometric methods respectively. composes a list of authentication service methods which it 

Dispatch Layer 154 comprises a set of callback functions. supports. This list may be ordered by preference, such that 

which facilitate connections to services. These functions jf a user prefers a particular method, it appears at the front 

allocate and initialize data structures used by the 0 f the list The client then sends its list to the server. In step 

connections, initialize connections, provide information x 168. the server compares the client's list with its own list to 

about security services, and terminate connections to ser- determine jf the processes share an authentication method, 

vices. The dispatch layer calls each of these functions. The server then sends back one of the following responses: 

In the initialization phase, the dispatch layer cycles (1) an error number — indicating that an error occurred 

through an array of service entries calling the information during the search; (2) authentication required — indicating 

function for each. After the information functions are called, 55 that the server does not support any of the methods the client 

the dispatch layer calls the initialization function for each specified, and authentication is required by the server; (3) 

service. The initialization function returns which adapters, if authentication type — describes the authentication method to 

any, are linked in for the service and whether or not the be used. The server ordinarily uses Che first authentication 

service is required. method it supports in the client's list; or (4) no 

In the connection phase, the supervisor similarly cycles 60 authentication — indicating the client and server could not 

through the array of service entries, repeatedly calling agree on an authentication service method, however, authen- 

services. One difference between the connect function for tication is not required. If agreement has been reached on an 

the client and the connect function for the server is that on authentication type, the server calls the initialization and 

the first call, the client sends information while the server function of the adapter corresponding to the agreed-upon 

waits for information. 65 authentication method. 

After the connection phase is complete, some services. If the negotiation fails, the client aborts the connection as 

such as encryption security service 122 and data integrity shown in step 168. Note that no failure occurs if authenti- 
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cation is not required by either process, even though the 
server docs not support any authentication services in com- 
mon with the client 

If the negotiation does not fail and authentication will be 
used, in step 170. the client: (1) calls the initialization 
function of the agreed-upon authentication method; (2) calls 
the connect function for the authentication method; (3) 
obtains an authentication credential using the retrieval func- 
tion of the authentication adapter; and (4) sends the creden- 
tial to the server. 

In step 172. upon response from the client, the server: (1) 
initializes its context using the agreed-upon authentication 
method; (2) stores the data about the client into its authen- 
tication context; and (3) calls the validation function of the 
authentication adapter. 

The next action taken by the server depends upon the 
status returned by the validation function of the authentica- 
tion adapter. If the status returned is "successful," the server 
returns that status and other information to the client If the 
status returned is "negotiation not complete,*' the server 
returns to the client a "more authentication required** status 
and other data specific to the authentication method. This 
status may occur, for example, if the particular authentica- 
tion method chosen requires more than one authentication 
call. If the status returned is 'failure." the server simply 
returns the failure status to the client. 

Next, in step 174, the client performs one of the following 
actions depending on the server response in step 172. If the 
server indicates "success," the client stores any return data 
in the authentication structure. If the server indicates "more 
authentication", the client uses the returned data, if any, to 
repeat the connection process as illustrated by the loop 
arrow in FIG. 5. If the server indicates 'failure,** the client 
aborts the connection. 

After the connect protocol is complete, the "connection 
completion** routine is called. If it was decided during the 
negotiation that no external authentication is to be used, 
resources used by the authentication method will be 
released. 

According to preferred embodiments of the invention, the 
negotiation process for security services other than 
authentication, such as encryption and checks urriming are 
handled differently than the way authentication is handled. 
According to one ernbodirnent, the negotiation process for 
encryption or checksumming may be divided into two parts. 
The first part determines whether a particular service such as 
encryption or checksumming will be used. If the service will 
be used, the second part of the negotiation determines which 
method of providing the service will be used. 

FIG. 6 illustrates the first part of the negotiation process, 
according to one embodiment of the invention. As shown, in 
steps 270 and 272. both the client and the server generate 
preference values. Preference values, as will be further 
discussed below, represent particular preferences for using a 
particular security service. In step 274, the preference values 
are compared- As a result of the comparison in step 274. one 
of three outcomes may result The communication may fail, 
as shown in step 276. If the communication does not fail, the 
security service may either be activated, as shown in step 
278, or the communication may proceed without the service 
being used as shown by arrow 280. In any case the server 
ordinarily sends a response back to the client indicating the 
outcome of this part of the negotiation process. 

Ordinarily, each party will at least have two options in 
expressing its preference for the service in question: a 
•required'* value which indicated that the service is required 
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for the communication; and a 'rejected" value which indi- 
cates that the service cannot be used by the party. Addition- 
ally according to the invention, one or more additional 
possible values are provided to at least one party. Adding 

5 additional intermediate values beyond "rejected" and 
"required" advantageously helps prevent needless commu- 
nication failures due to one party specifying ''required** or 
"rejected" when the service was actually only preferred, and 
also advantageously helps prevent agreement on a service or 

io algorithm which is less than ootimaL 

Thus, according to one ernbodirnent of the invention, at 
least one party is provided with an additional value. The 
additional value may be called a "don't care** value, which 
indicates mat the particular service is neither "rejected" nor 

15 "required", and that either using the service or not would be 
satisfactory. 

FIG. 7 is a table of outcomes of this part of the negotiation 
process according to one such embodiment. Note that if the 
client specifies the ''Don't care** value, the security service 
20 in question will be used if the server requires it, and the 
service will not be used if the server rejects it. 

According to another embodiment of the invention, both 

the client and the server are able choose from four possible 

preference values. 
25 v 



Value Meaniqg 

"required" Indicates that a service is required for 

the communkatkn. The service must be 
active, and die connection wilt fail if 
the other side specifies "rejected" or if 
there is no coinp alible algorithm on the 
other side. 

"requested* hvft-atm thai a service is not 

essential, but is preferred The service 

35 will be active if the other side 

specifies "accepteo* \ "requested" » or 
"required*' and there is a compatible 
algorithm available on the other side; it 
will be inactive otherwise, 
"accepted" Tnrtfratrn that a service is not 

40 essential, but is not preferred. The 

service will be active if the other side 
of the connection specifies "requester 
or "roqiiire<r' and there is a compatible 
algorithm available on Che other side; it 
wiU be inactive otherwise. 

45 "rejected" TiTtr**** that the service must not be 

used, and (he connection will fail if the 
other side specifies "required." 



If one party fails to specify a preference value for a 

50 particular service, a value of "accepted** may be assigned by 
default if that service is available, or a default value of 
"rejected" may be assigned if that service is not available. 

FIG. 8 presents the outcomes for the possible combina- 
tions of client and server preferences, according to an 

55 embodiment of the invention in which each party's prefer- 
ence has the above four possible values. If the outcome is 
"fail** then the communication will fall since one side 
specified the service is required, and the other side specified 
it is rejected. If the outcome is **ofT\ the service will not be 

60 activated, but the connection will still be made. If the 
outcome is **on*\ the service will be activated if the two sides 
of the connection have an algorithm in common. If the 
outcome is "on" and two sides do not have an algorithm in 
common the communication will fail if either side specified 

63 a ^equiTed" value. If there was no algorithm in common, but 
neither side of the connection specified "required**, then the 
service will not be activated and the connection will be 
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made. Although the different preference values have been According to one embodiment of the invention, an addi- 

described in terms "required", "requested", "accepted", and tional field may be encoded and included with the negotia- 

"rejected , ^ many other labels may be used to indicate tion information being transmitted between the client and 

different levels of preference. Additionally, in general, any server when a service such as encryption is activated: a 

number of possible preference values may be provided to 5 "sanity cheer value. This value may be the result of one 

either side of the connection. encrypting a fixed quantity known in advance by both 

, ends of the connection, which may be hard-coded into the 

FIG. 9 illustrates the second part of the negotiation software . Th e othcr ^ upon receiving the sanity check 

process, which selects a particular method or algorithm of vjduc ^ compare tf, e reS ulting value to the value with the 

providing a service. In step 180. the client and server both cncryp ti 0I1 algorithm and will signal an error if the 

generate lists of service methods they are willing to use. In 10 ^ encrypted valoes do not match, 

general, step 180 may be done at any time. For example the According to another embodiment of the invention, both 

server may create a single list during its own set up ^ ^ ^ %awa ^ ^ tQ activatc „ deactivate 

procedure, or it may create a separate list for each e Uon checksumming over the course of a 

connection, depending on the identity of toe client. Prefer- a,^^ but eilhcr sJdc wil i be able to stipulate, when the 

ably the lists are ranked, such that a party s most preferred « section is opened, that deactivation is forbidden. The 

algorithm i or method is first on the list. In step 182. the client, ne(WQrk ^ ^ rcs ^ nsiblc for enforcing Qfo 

server or both transmit mar preference lists to the other side. re)luiremellt M ^ no application program may be allowed 

In step 184. the lists are compared to determine if a t0 The abffity to activate and deacdvatealgoriAms 

particular method can be agreed upon. If the lists are in a d . ^ rf a ooonBC|ioB advan tageously allows for 

prioritized order, the comparison in step 184 may also select » ^ lament # data such as password changes, 

the highest priority method or algorithm rf more than one ^ encryption on other data. This may be 

match is found. As is indicated by the dashed arrow 186 in desirable ^ vea ^ some encryption algorithms are slow. 

FIG. 9. the process of generating and comparing preference ^ oAas canno , ^ e3tported cxcepl for use ta protecting 

lists may repeat multiple times before a service method is passwords 

agreed upon, or until it is determined that no agreement is * ^ g embodimen , of ^ ^tion. application 

P° sslWc * programs may be able to control the use of encryption on a 

According to a preferred embodiment of the invention. connection through connect-time settings and later, once the 
both parts of the negotiation process may advantageously be coimect i on has been established, by means of runtime con- 
combined into a single exchange of preferences, thereby ^ troL At connect time, the application should be able to 
simplifying and expediting the negotiation process. The control whether cncry ption: (1) is to be available at all; (2) 
preference value selected in the first part of the negotiation is to ^ acdve when mc connection first opens; (3) may be 
process can be effectively incorporated into the list of deactivated during the course of the connection; and (4) is 
preferred service methods by using a NULL value to indi- automatically activated or deactivated when the other side 
cate the preference for not using any algorithm for the ^ activated or deactivated it. respectively. This feature advan- 
security service. In other words, the NULL value indicates tageously makes it possible for one end of the connection to 
a preference not to use the security service at all A list sen(J encrypted data while the other end sends unencrypted 
consisting solely of a NULL value Indicates a service is ^ 0nce ^ connection is established, each application 
Rejected.- A list containing service methods, but no NULL may ^ ^ t0 : (1) deactivate encryption, subject to other 
value, indicates that a service is 'Quired" A list containing ^ requirements; and (2) reactivate encryption. Alternatively, 
a NULL value in the first position indicates a service is another embodiment might only give the ability to activate 
'^referred" Finally, a list containing a NULL value in the ^ restore encryption if encryption was activated when the 
last position indicates a service is "accepted." connection was opened. 

If the client side of the connection fails to specify a set of Although the foregoing embodiments have been 

methods or algorithms for a particular service, aU methods 45 described with reference to two nodes, namely a client and 

available on the client side may be considered acceptable as a serV er, the invention is also applicable to negotiations 

a default condition. Additionally, if the server side fails to between more than two nodes. For example, a three node 

specify a list of methods, the process may revert to using a negotiation is possible, and in a similar fashion to the 

predetermined default order of preference. two-node embodiments, each node would select its own 

According to a preferred embodiment of the invention, 50 preference value and lists of algorithms for a particular 

FIG. 1# illustrates a scheme which may be used to decide service; transfer the information to at least one of the nodes, 

whether a service will activate. In step 200 the preference Upon comparing the preferences and lists, it would be 

lists are compared using the table in FIG. 8. In step 202, if determined whether or not to use the service, and if so a 

the result is "fail", the communication will not proceed If particular algorithm would be selected, 

the result is "off", as indicated by arrow 2*4, the service will 55 According to a further preferred embodiment perfor- 

Dot activate, but communication will proceed. mance may advantageously be enhanced in several ways : ( 1) 

Several outcomes arc possible if the result is "on." In step the mechanism which checks whether a service is to be used 

206, the lists are compared to determine if there is a service may be implemented through macros instead of functions to 

algorithm in common. In step 208, if the two sides have a reduce the amount of time wasted by unnecessary function 

service algorithm in common, the service will be active and 60 calls when no service is enabled; (2) negotiations for ser- 

the communication will proceed using an algorithm in vices such as authentication, encryption and checksumming 

common. In step 210. if the two sides do not have a service may be combined into the same packets, so that the total 

method in common, the outcome depends on whether either number of packet transfers required to negotiate all security 

side specified "required." In step 212. if either side specified services is reduced; and (3) data buffers and packet lengths 

Required," the connection will fail. In step 214. if neither 65 may be expanded, or structured to be expandable to accom- 

side specifics "required." the service will be inactive and modate the differing amounts of data expansion arising from 

communication will proceed. different encryption and checksumming methods. 
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According to a preferred embodiment of the invention. determining that communication between the first and 

FIG. 11 illustrates packet format 229 which provides for second nodes should not proceed when one of the 

expanded data packet lengths. Packet format 220 is illustra- preference values includes the value representing that 

tive of a situation where both encryption and checksumming the security service is required and the other preference 

are used. As shown in FIG. 11. packet format 220 comprises 5 value includes the value representing that the security 

a header field 222. encrypted data field 224. length of service is rejected. 

unencrypted data 226. and cryptographic checksum 228. 2. The process of claim 1. further comprising the steps of: 

Header field 222 is shown being unencrypted, while data creating in the first node a first preference list; 

fields 224. 226 and 228 are shown as being encrypted. Field creating in the second node a second preference list, the 

226 contains, in encrypted form, the length of the unen- J0 f ast ^ second preference lists including one or more 

crypted data. According to this embodiment, field 226 is acceptable methods for the security service; 

included in the packet whenever the dau has been com^^g t^ccn merlrst and second node at least 

encrypted. It is not necessary to include, in field 226. the , . ° . „_„. „„ f „„, „ ... . 

unencrypted length of the cryptographic checksum 228. I one of ^ **** s f ond pre f f erence ^ and . 

since that ordinarily has a fixed leiglfa for each algorithm. xlt ***S a mcthod *»«he purity servl « by comparing 

regardless of the size of the data. In order to encode the 13 , ** preference list with the second preference list 

length of the unencrypted data in a fixed-length section of ?• The process of chum 2. wherein the first preference 

the packet such as field 226. the maximum amount of data valuc 15 included m the first preference list and the second 

that can ever appear in a packet needs to be known. If a fixed preference valuc is included in the second preference list 

upper bound onthe amount of expansion expected for any 4 ™* P"*** 5 of «*■«■« 3- wl « reu > the i . firs "; Preference 

encryption algorithm is known, it is possible to encode the 20 valuc * lncluded ln me P«f«"»ce by selecting 

difference between the sizes of the encrypted and unen- members of the first preference list from a set which includes 

crypted data. Note that if a driver for an encryption algo- a nuU value - and by ordering the preference list according to 

rithm never expands the data, field 226 may be omitted P**y wherein the null value represents not using the 

altogether security service, and the position of the null value in the 

FIG. 12 illustrates how breaks in communication are 25 preference list indicates ithe ftost preference value, 

handled when a security service is active. In step 230. the „ 5 - P^*™ 5 of claim 1 further comprising ftestep of 

system first checks tosei if a security service is active. If so. d^hvatu* to security service m response to an instruction 

the security service is reset in step 232. In step 234. a new one of the first or second nodes 

key is produced from the old key through a^edetermined ^f 0 ^ * C *T * COmpnS1 ° 8 

function at both server and client One such predetermined 30 ^activating the .security semce in response to an instruc- 

... , . ^_ 4 . . j * • j tion from one of the first or second nodes. 

function is to exclusive-OR the key with a predetermined " ™r . . 4 . ^ • — * 

random bit pattern. In step 236 the convocation is 7. The process of claim 1, furtha comprmng the steps of: 

re-started In this way. both sides may re-synchronize with- encrypting a set of data and the length of the set of data; 

out communication. creating a data packet including the encrypted form of the 

We claim: 35 sct °* < * ata ^ ^ len 8 m °* me sct °f c * ata i ^ 

1. In a networked computer environment a process for communicating the data packet between the first and 

determining whether a security service will be used for second nodes. 

communication between a first node and a second node. 8. The process of claim 1, wherein the security service is 

comprising the steps of: an authentication process and further comprising the steps 

selecting in the first node, a first preference value for the 40 °f : 

security service; cornmunicating from the first node to the second node 

selecting in the second node, a second preference value authentication data; 

for the security service, at least one of the first or verifying in the second node the identity of the first node 

second preference values being selected from a set of 4J using the authentication process; and 

values comprising a value representing that the security comrnunicating from the second node to the first node a 

service is required, a value representing that the secu- response representing the result of said step of verify- 

rity service is rejected, a first intermediate value rep- ing. 

resenting that the security service is preferred but is not 9. In a networked computer environment including mul- 

essential, and a second intermediate value representing ^ tiple nodes, a process for selecting whether a security service 

that the security service is not preferred but will be will be used during communication between a first node and 

tolerated; a second node and for selecting a method for the security 

communicating between the first and second node at least service, said process comprising the steps of: 

one of the first or second preference values; creating in the first node a first preference list; 

comparing the first preference value with the second 55 creating in the second node a second preference list, the 

prefaence value; first and second prefaence lists including acceptable 

determining that the security service will be used when methods for the security service, each of the first and 

neither the first or second prefaence values represent second prefaence lists being ordaed by priority and 

that the security service is rejected and both preference includes a null value when the security service is not 

values do not include the second intermediate value; 60 essential, wherein the null value represents not using 

toermining that the security service will not be used and «e security service, and the position of the null value 

that communication between the first and second nodes in each ordered prefaence list indicates strength of 

should proceed when both prefaence values include prefaence for not using the security service; 

the second intermediate value, or when one preference communicating between the first and second node at least 

value represents that the security service is rejected and 65 one of the first and second preference lists; 

the other preference value does not represent that the comparing the first preference list with the second pref- 

security service is required; and erence list; 



08/15/2003, EAST 



Version: 1 . 04 . 0000 



5,7J 

13 

determining that the security service will be used when 
both the first and second preference lists include meth- 
ods other than the null value, both preference lists do 
not indicate that the security service is not preferred but 
will be tolerated, and a match is found between the first 
and second preference lists; and 

selecting a method for the security service by selecting a 
method contained in both preference lists. 

10. The process of claim 9. wherein said step of selecting 
a method comprises dctennining that no further com muni- 
cation should take place between the first and second nodes 
when no match is found between the first preference list and 
the second preference list 

11. An apparatus for determining whether a security 
service will be used in a networked computer environment 
comprising: 

a first node configured to select a first preference value for 
the security service; 

a second node configured to select a second preference 
value for the security service; at least one of the first or 
second preference values being selected from a set 
comprising a value representing that the security ser- 
vice is required* a value representing that the security 
service is rejected, a first intermediate value represent- 
ing that the security service is preferred but is not 
essential, and a second intermediate value representing 
that the security service is not preferred but will be 
tolerated, said second node further configured to 
receive the first preference value and determine 
whether the security service will be used by comparing 
the first preference value with the second preference 
value, to determine that the security service will be 
used when neither the first or second preference values 
represents that the security service is rejected and both 
preference values do not include the second interme- 
diate value, to determine that the security service will 
not be used and that communication between the first 
and second nodes should proceed when both preference 
values include the second intermediate value, or when 
one preference value represents that the security service 
is rejected and the other preference value does not 
represent that the security service is required, and to 
determine that communication between the first and 
second nodes should not proceed when one of the 
preference values includes the value representing that 
the security service is required and the other preference 
value includes the value representing that the security 
service is rejected; and 

a network connecting said first node and said second 
node, and adapted to transmit from said first node to 
said second node the first preference value. 

12. The apparatus of claim 11. wherein: 

the first node is further configured to create a first pref- 
erence list including one or more acceptable methods 
for the security service, the first preference list includ- 
ing the first preference value; 

the second node is further configured to create a second 
preference list including one or more acceptable meth- 
ods for the security service, the second preference list 
including the second preference value; and 

the second node is further configured to receive the first 
preference list from the network and to select a method 
for the security service by comparing the first prefer- 
ence list with the second preference list 

13. The apparatus of daim 12. wherein the first preference 
value is included in the first preference list by selecting 
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members of the first preference list from a set which includes 
a null value, and by crdcring the first preference list accord- 
ing to priority, and wherein the null value represents not 
using the security service, and the position of the null value 
5 in the ordered preference list indicates the first preference 
value. 

14. A computer-readable medium which can be used to 
direct a computer node in a networked computer environ- 
ment to determine whether a security service will be used for 

to communication between a first node and a second node, 
comprising: 

means for directing the first node to select a first prefer- 
ence value; 

means for directing a second node to select a second 

15 preference value, at least one of the first or second 
preference values being selected from a set comprising 
a value representing that the security service is 
required, a value representing that the security service 
is rejected, a first intermediate value representing that 

20 the security service is preferred but is not essential, and 
a second intermediate value representing that the secu- 
rity service is not preferred but will be tolerated; 
means for directing the first node to transmit the first 
preference value to the second node; 

25 means for directing the second node to determine whether 
the security service will be used by comparing the first 
preference value with the second preference value, to 
determine that the security service will be used when 
neither the first or second preference values represent 

30 that the security service is rejected and both preference 
values do not include the second intermediate value, 
and to determine that the security service will not be 
used and that communication between the first and 
second nodes should proceed when both preference 

35 values include the second intermediate value, or when 
one preference value represents that the security service 
is rejected and the other preference value does not 
represent that the security service is required; and 

^ means for directing the second node to find that commu- 
nication between the first and second nodes should not 
proceed when one of the preference values includes the 
value representing that the security service is required 
and the other Preference value includes the value 

4J representing that the security service is rejected. 

15. The computer-readable medium of claim 14, further 
comprising: 

means for directing the first node to create a first prefer- 
ence list including one or more acceptable methods for 
yj the security service, the first preference list including 
the first preference value; 
means for directing the second node to create a second 
preference list including one or more acceptable meth- 
ods for the security service, the second preference list 
55 Including the second preference value; 

means for directing the network to transmit the first 

preference list to the second node; and 
means for directing the second node to select a method for 
the security service by comparing the first preference 
60 List with the second preference list 

16. The apparatus of claim IS. wherein the first preference 
value is included in the first preference list by selecting 
members of the first preference list from a set which includes 
a null value, and by ordering the preference list according to 

65 priority, and wherein the null value represents not using the 
security service, and the position of the null value in the 
ordered preference list indicates the first preference value. 
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17. The process of claim 16, wherein (he first preference 
value represents that the security service is required when 
the null value is not present in the preference list, the first 
preference value represents that the security service is 
rejected when the null value is the only method present in 5 
the preference list, the first preference value is the first 
intermediate value when the null value is positioned as the 
highest priority method in the preference list and the first 
preference value is the second intermediate value when the 
null value is positioned as the lowest priority method in the 10 
preference list. 

18. The process of claim 4, wherein the first preference 
value represents that the security service is required when 
the null value is not present in the preference list, the first 
preference value represents that the security service is 15 
rejected when the null value is the only method present in 
the preference list the first preference value is the first 
intermediate value when the null value is positioned as the 
highest priority method in the preference list and the first 
preference value is the second intermediate value when the 20 
null value is positioned as the lowest priority method in the 
preference list, 

19. The process of claim 18 wherein the security service 
is encryption and the methods for the security service 
comprise algorithms for encryption. 25 

20. The process of claim 9 wherein the first preference list 
includes the null value and includes no other methods for the 
security service when the security service must not be used 
by the first node. 

21. The process of claim 20 wherein the first preference 30 
list includes the null value in the highest priority position 
and includes one or more methods for the security service in 
lower priority positions when the security service is not 
essential but is preferred by the first node. 

22. The process of claim 21 wherein the first preference 35 
list includes the null value in the lowest priority position and 
includes one or more methods for the security service in 
higher priority positions when the security service is not 
preferred but will be tolerated by the first node. 

23. The process of claim 22 wherein the security service 40 
is encryption and the methods for the security service 
comprise algorithms for encryption. 

24. The apparatus of claim 13 wherein the first preference 
value represents that the security service is required when 
the null value is not present in the preference list the first 45 
preference value represents that the security service is 
rejected when the null value is the only method present in 
the preference list the first preference value is the first 
intermediate value when the null value is positioned as the 
highest priority method in the preference list and the first 50 
preference value is the second intermediate value when the 
null value is positioned as the lowest priority method in the 
preference list. 

25. An apparatus for determining whether a security 
service will be used in a networked computer environment 55 
comprising: 

a first node configured to create a first preference list; 

a second node configured to create a second preference 
list, wherein the first and second preference lists 
include acceptable methods for the security service, 
each of the first and second preference lists being 
ordered by priority and includes a null value when the 
security service is not essential, the null value repre- 
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seating not using the security service, and the position 
of the null value in each ordered preference list indi- 
cates strength of preference for not using the security 
service; and 

a network connecting said first and second nodes, and 
adapted to communicate between the first and second 
node at least one of the first and second preference lists, 
wherein said second node is further configured to 
compare the first preference list with the second pref- 
erence list, to determine that the security service will be 
used when both the first and second preference lists 
include methods other than the null value, both pref- 
erence lists do not indicate that the security service is 
not preferred but will be tolerated, and a match is found 
between the first and second preference lists; and to 
select a method for the security service by selecting a 
method contained in both preference lists. 

26. The apparatus of claim 25. wherein said second node 
is further configured to determine that no further commu- 
nication should take place between the first and second 
nodes when no match is found between the first preference 
list and the second preference list. 

27. A computer-readable medium which can be used to 
direct a computer node in a networked computer environ- 
ment to determine whether a security service will be used 
during communication between a first node and a second 
node and to select a method for the security service, com- 
prising: 

means for directing the first node to create a first prefer- 
ence list; 

means for directing the second node to create a second 
preference list, the first and second preference lists 
including acceptable methods for the security service, 
each of the first and second preference lists being 
ordered by priority and includes a null value when the 
security service is not essential, wherein the null value 
represents not using the security service, and the posi- 
tion of the null value in each ordered preference list 
indicates strength of preference for not using the secu- 
rity service; 

means for directing the first node to communicate the first 

preference list to the second node; 
means for directing the second node to compare the first 

preference list with the second preference list; 
means for directing the second node to determine that the 

security service will be used when both the first and 

second preference lists include methods other than the 

null value, both preference lists do not indicate that the 

security service is not preferred but will be tolerated. 

and a match is found between the first and second 

preference lists; and 
means for directing the second node to select a method for 

the security service by selecting a method contained in 

both preference lists. 

28. The computer-readable medium of claim 27 further 
comprising means for directing the second node to deter- 
mine that no further communication should take place 
between the first and second nodes when no match is found 
between the first preference list and the second preference 
list 
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